feat(local): ask a local checkpoint its options by letter - #102
Merged
Merged
Conversation
The local arm read each option's own token and refused options that are several tokens, so on the public fixture it scored 39 of 105 rows. --option-style letter lists the options as A, B, C and reads the letter tokens: every row scores, the distribution still maps back to the labels, and the artifact and the report say the question was lettered. The style joins the cache key only when it is letter, so existing entries keep theirs. Run for real, the pinned Qwen2.5-1.5B landed at chance accuracy with an ECE of 0.349 against a floor of 0.087. Sequence probability was the alternative and was not taken: longer options lose probability for being long, and any length correction is a choice the result would silently depend on. METHODOLOGY says so. Part of #3. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3. This implements the decision on multi-token options: letter labels, opt-in.
What it does.
--option-style letter(the adapter argument isoption_style="letter") lists the options asA. pay_full_estimate, B. deny_vacancy_exclusion, …and reads the logits of the single-token letters.label, which reads each option's own token and refuses multi-token options. Its refusal message now suggests letter mode.Recorded everywhere.
call_paramsincludesoption_styleonly in letter mode, so every existing label-mode cache key is unchanged (tested). Letter mode already keys on option order, sincelabel_order_mattersholds for this arm.rawcarriesoption_styleand the label-to-letter map.option_style.Run for real. Pinned Qwen2.5-1.5B-Instruct on the GPU, over the public fixture:
With every row scored, the null tells you something it couldn't before: the model is at chance accuracy but confident, and that miscalibration is clearly distinguishable from noise.
Why not sequence probability? Longer options would lose probability for being long, and every length correction is a choice the result would silently depend on. METHODOLOGY's
restricted_softmaxsection now covers both readings and this reasoning.Tests cover multi-token options asked by letter, the default letter instructions, more than 26 options refused, the cache key moving only in letter mode, an unknown style refused, and the CLI flag reaching the local arm while the mock refuses it.
docs/example-report.mdis unchanged. The gate, the site checks and the prose checks all pass.🤖 Generated with Claude Code